Reenable incremental builds #162

james-bruten-mo · 2026-01-20T14:50:27Z

PR Summary

Sci/Tech Reviewer: @MatthewHambley
Code Reviewer: @t00sa

At the git migration incremental builds on the command line no longer worked as timestamps in github are based on clone date, not commit date. This reenables this functionality by fetching/pulling changes if the build is being rerun. This has been tested on the command line for both a run with physics code (lfric_atm) and without (gungho_model). The 2nd build time is much quicker as expected.

This also enables building using the local github mirrors which will be useful once these are also available on the HPC. Again, these have been tested on the cli and with incremental builds.

This change adds get_git_sources which is mostly a copy of the same file from SimSys_Scripts. Having 2 copies isn't ideal, but I can't think of another way to make that SimSys_Scripts file available when building locally (short of installing it as a library). Hopefully this is something that'll be improved by fab!

Code Quality Checklist

I have performed a self-review of my own code
My code follows the project's style guidelines
Comments have been included that aid understanding and enhance the readability of the code
My changes generate no new warnings
All automated checks in the CI pipeline have completed successfully

Testing

I have tested this change locally, using the LFRic Core rose-stem suite
If required (e.g. API changes) I have also run the LFRic Apps test suite using this branch
If any tests fail (rose-stem or CI) the reason is understood and acceptable (e.g. kgo changes)
I have added tests to cover new functionality as appropriate (e.g. system tests, unit tests, etc.)
Any new tests have been assigned an appropriate amount of compute resource and have been allocated to an appropriate testing group (i.e. the developer tests are for jobs which use a small amount of compute resource and complete in a matter of minutes)

trac.log

Test Suite Results - lfric_apps - test_incremental_builds_change/run2

Suite Information

Item	Value
Suite Name	test_incremental_builds_change/run2
Suite User	james.bruten
Workflow Start	2026-01-20T15:03:56
Groups Run	developer

Dependency	Reference	Main Like
casim	MetOffice/casim@2025.12.1	True
jules	MetOffice/jules@2025.12.1	True
lfric_apps	james-bruten-mo/lfric_apps@improve_local_builds	False
lfric_core	MetOffice/lfric_core@aa32824	True
moci	MetOffice/moci@2025.12.1	True
SimSys_Scripts	MetOffice/SimSys_Scripts@2025.12.1	True
socrates	MetOffice/socrates@2025.12.1	True
socrates-spectral	MetOffice/socrates-spectral@2025.12.1	True
ukca	MetOffice/ukca@2025.12.1	True

Task Information

✅ succeeded tasks - 1106

Security Considerations

I have reviewed my changes for potential security issues
Sensitive data is properly handled (if applicable)
Authentication and authorisation are properly implemented (if applicable)

Performance Impact

Performance of the code has been considered and, if applicable, suitable performance measurements have been conducted

AI Assistance and Attribution

Some of the content of this change has been produced with the assistance of Generative AI tool name (e.g., Met Office Github Copilot Enterprise, Github Copilot Personal, ChatGPT GPT-4, etc) and I have followed the Simulation Systems AI policy (including attribution labels)

Documentation

Where appropriate I have updated documentation related to this change and confirmed that it builds correctly

PSyclone Approval

If you have edited any PSyclone-related code (e.g. PSyKAl-lite, Kernel interface, optimisation scripts, LFRic data structure code) then please contact the TCD Team

Sci/Tech Review

I understand this area of code and the changes being added
The proposed changes correspond to the pull request description
Documentation is sufficient (do documentation papers need updating)
Sufficient testing has been completed

(Please alert the code reviewer via a tag when you have approved the SR)

Code Review

All dependencies have been resolved
Related Issues have been properly linked and addressed
CLA compliance has been confirmed
Code quality standards have been met
Tests are adequate and have passed
Documentation is complete and accurate
Security considerations have been addressed
Performance impact is acceptable

MatthewHambley

Some improvements possible to structure and code. Also some lack of clarity regarding why certain things are done in certain ways.

build/extract/extract_science.py

build/extract/get_git_sources.py

MatthewHambley · 2026-01-21T15:47:51Z

build/extract/get_git_sources.py

+        commands = (
+            f"git -C {loc} init",
+            f"git -C {loc} remote add origin {repo_source}",
+            f"git -C {loc} fetch origin {repo_ref}",
+            f"git -C {loc} checkout FETCH_HEAD",
+            f"git -C {loc} fetch origin main:main",
+        )


This seems a lot of effort. Doesn't it just recreate a clone?

Yes, it was initially written to see if we could save space while cloning sources for use in rose-stem. It turns out the difference is very small, but as this was marginally smaller, I left it in. Given it's copied from SimSys_Scripts I don't think it's worth modifying here

In that case a comment saying that it is exactly equivalent to "clone" will be welcome by anyone trying to understand why all this is happening.

MatthewHambley · 2026-01-21T15:51:10Z

build/extract/get_git_sources.py

+    # Fetch the main branch from origin
+    # Ignore errors - these are likely because the main branch already exists
+    # Instead write them as warnings
+    command = f"git -C {loc} fetch origin main:main"


If you need to treat the copy as a repository, why not clone it. That's what distributed repositories are all about. You might take advantage of a sparse clone to gain the advantage of filtering you are getting from the rsync exclusion list.

build/local_build.py

documentation/source/developer_guide/local_builds.rst

…mo/lfric_apps into improve_local_builds

james-bruten-mo

Thanks Matthew - those comments are addressed, and for the most part changes done

build/extract/extract_science.py

build/extract/get_git_sources.py

james-bruten-mo · 2026-01-21T16:45:51Z

build/extract/get_git_sources.py

+        commands = (
+            f"git -C {loc} init",
+            f"git -C {loc} remote add origin {repo_source}",
+            f"git -C {loc} fetch origin {repo_ref}",
+            f"git -C {loc} checkout FETCH_HEAD",
+            f"git -C {loc} fetch origin main:main",
+        )


Yes, it was initially written to see if we could save space while cloning sources for use in rose-stem. It turns out the difference is very small, but as this was marginally smaller, I left it in. Given it's copied from SimSys_Scripts I don't think it's worth modifying here

build/local_build.py

documentation/source/developer_guide/local_builds.rst

build/local_build.py

MatthewHambley

A few outstanding conversations and a few new thoughts.

build/extract/extract_science.py

MatthewHambley · 2026-01-22T13:37:23Z

build/extract/get_git_sources.py

+        commands = (
+            f"git -C {loc} init",
+            f"git -C {loc} remote add origin {repo_source}",
+            f"git -C {loc} fetch origin {repo_ref}",
+            f"git -C {loc} checkout FETCH_HEAD",
+            f"git -C {loc} fetch origin main:main",
+        )


In that case a comment saying that it is exactly equivalent to "clone" will be welcome by anyone trying to understand why all this is happening.

Co-authored-by: Matthew Hambley <MatthewHambley@users.noreply.github.com>

james-bruten-mo added 4 commits January 19, 2026 15:57

incremenental with core working

1c35728

incremental physics builds

cc1d00d

Merge remote-tracking branch 'upstream/main' into improve_local_builds

137ba5f

incremental builds

132ccf8

github-actions bot added the cla-signed The CLA has been signed as part of this PR - added by GA label Jan 20, 2026

github-actions bot assigned james-bruten-mo Jan 20, 2026

james-bruten-mo added 3 commits January 20, 2026 14:57

Merge remote-tracking branch 'upstream/main' into improve_local_builds

90d113d

minor changes

82f3795

catch missing env var

35903fb

james-bruten-mo removed the cla-signed The CLA has been signed as part of this PR - added by GA label Jan 20, 2026

james-bruten-mo marked this pull request as ready for review January 20, 2026 20:59

github-actions bot requested a review from MatthewHambley January 20, 2026 21:00

james-bruten-mo mentioned this pull request Jan 21, 2026

extract_science.py rebuild fix #95

Closed

28 tasks

update local build docs

b6137e1

github-actions bot added the cla-modified The CLA has been modified as part of this PR - added by GA label Jan 21, 2026

Merge branch 'main' into improve_local_builds

8e9c249

james-bruten-mo removed the cla-modified The CLA has been modified as part of this PR - added by GA label Jan 21, 2026

MatthewHambley requested changes Jan 21, 2026

View reviewed changes

james-bruten-mo added 2 commits January 21, 2026 17:06

review comments

7500ed6

Merge branch 'improve_local_builds' of ssh://github.com/james-bruten-…

c59a673

…mo/lfric_apps into improve_local_builds

james-bruten-mo commented Jan 21, 2026

View reviewed changes

james-bruten-mo requested a review from MatthewHambley January 21, 2026 17:07

MatthewHambley requested changes Jan 22, 2026

View reviewed changes

Update build/extract/extract_science.py

f48745e

Co-authored-by: Matthew Hambley <MatthewHambley@users.noreply.github.com>

Reenable incremental builds #162

Are you sure you want to change the base?

Reenable incremental builds #162

Uh oh!

Conversation

james-bruten-mo commented Jan 20, 2026 • edited by MatthewHambley Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Code Quality Checklist

Testing

trac.log

Test Suite Results - lfric_apps - test_incremental_builds_change/run2

Suite Information

Task Information

Security Considerations

Performance Impact

AI Assistance and Attribution

Documentation

PSyclone Approval

Sci/Tech Review

Code Review

Uh oh!

MatthewHambley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MatthewHambley Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

james-bruten-mo Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

MatthewHambley Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

MatthewHambley Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

james-bruten-mo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

james-bruten-mo Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MatthewHambley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MatthewHambley Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

james-bruten-mo commented Jan 20, 2026 •

edited by MatthewHambley

Loading